Introduction to Machine Learning

Chapter 16: Logistic Regression

1. Introduction

Linear regression predicts unbounded real numbers — perfect for house prices or temperature, but catastrophically wrong for classification, where we want a valid probability in the range [0, 1]. This unit introduces Logistic Regression, the workhorse of binary classification. Despite the name, it is a classification model: it squashes a linear predictor through the Sigmoid function to produce calibrated probabilities, then thresholds them to make class predictions. We derive the Binary Cross-Entropy (log-loss) cost function from maximum-likelihood principles, derive the elegant gradient-descent update rules, and work a full numerical example.

Learning Objectives

2. Theory

2.1 Why Linear Regression Fails for Classification

Task: Predict whether an Iris flower is Virginica (Y = 1) or not (Y = 0) from petal length + width. We try ordinary linear regression and get outputs like these:

\[ \hat{y}_{\text{linear}} \in \{-0.16,\ -0.03,\ 0.30,\ 0.46,\ 0.77,\ 0.92,\ 1.12,\ \ldots\} \]

Problems with linear regression for classification

  1. Invalid "probabilities": outputs can be negative or > 1. A probability must always satisfy \( 0 \le p \le 1 \).
  2. Wrong loss function: Squared error is statistically inappropriate for Bernoulli outcomes; it's also not the log-likelihood, so the resulting estimates don't have nice properties.
  3. Sensitive to outliers: A single far-away outlier can tilt the whole regression line and change classification thresholds completely.

We need a model that always outputs values in \( (0, 1) \) and has a probabilistic foundation. That model is logistic regression.

2.2 Logistic Regression: From Log-Odds to Probability

Logistic regression is used for classification, most commonly binary classification where \( Y \in \{0, 1\} \). We model a transformation of the probability linearly, then invert back.

Step 1: Define odds and log-odds (logit)

Let \( p = P(Y = 1 \mid x) \) be the probability of the positive class. The odds of success are:

\[ \text{Odds}(Y=1) = \frac{p}{1-p} \in [0,\ +\infty) \]

Take the natural log — now the range is the entire real line, perfect for a linear model:

\[ \text{logit}(p) = \ln\left( \frac{p}{1-p} \right) \in (-\infty,\ +\infty) \]

This is why it's called logistic regression: we perform a linear regression on the log-odds:

\[ \ln\left( \frac{p}{1-p} \right) = \theta_0 + \theta_1 x_1 + \cdots + \theta_q x_q = \theta^T x = z \]

Step 2: Invert to get probability — the Sigmoid

Solving for \( p \) by exponentiating both sides and rearranging gives us the Sigmoid (Logistic) function:

\[ p = \sigma(z) = \frac{1}{1 + e^{-z}} \in (0,\ 1) \]

The sigmoid "squashes" the unbounded linear output \( z = \theta^T x \) into the valid probability range \( (0, 1) \). Key properties:

Step 3: Make hard class predictions

Once we have \( p = \sigma(z) \), we apply a threshold function to get a binary prediction \( \hat{y} \):

\[ \hat{y} = \begin{cases} 1 & \text{if } \sigma(z) > 0.5 \\ 0 & \text{otherwise} \end{cases} \]

(The 0.5 threshold can be tuned — raise it to reduce false positives, lower it to reduce false negatives.)

2.3 Interpreting Coefficients

A fitted logistic-regression model for personal-loan acceptance (bank dataset, 5000 customers, only 9.6% accepted the previous campaign). The learned coefficients:

FeatureCoefficient \( \theta_j \)
Intercept (\( \theta_0 \))−4.0
Income (\( x_1 \), in $10K units)+0.8
Has Securities Account (\( x_2 \), 0/1)+1.2
Age (\( x_3 \), in years)+0.02

Worked Example: New Customer Prediction

Customer: Income = $70K → \( x_1 = 7 \), Has Securities → \( x_2 = 1 \), Age = 45 → \( x_3 = 45 \).

\begin{align} z &= \theta^T x = -4.0 + 0.8(7) + 1.2(1) + 0.02(45) \\ &= -4.0 + 5.6 + 1.2 + 0.9 = \mathbf{3.7} \end{align}

Now apply the sigmoid:

\[ p = \sigma(3.7) = \frac{1}{1 + e^{-3.7}} \approx \frac{1}{1 + 0.0247} \approx \mathbf{0.976} \]

The model estimates a 97.6% chance this customer will accept the loan — confidently predicted class = 1 (Accept).

Sign and magnitude intuition

2.4 Learning the Coefficients: Maximum Likelihood + Binary Cross-Entropy

Binary labels are Bernoulli trials. Each true label follows:

\[ y_i \sim \text{Bernoulli}(p_i), \quad p_i = P(y_i = 1 \mid x_i; \theta) = \sigma(\theta^T x_i) \]

The Bernoulli probability mass function can be written compactly in one line:

\[ P(y \mid x; \theta) = p^y (1-p)^{1-y} \]

Check: if \( y = 1 \), it gives \( p \); if \( y = 0 \), it gives \( 1-p \). ✓

Log-Likelihood of One Example

\[ \mathcal{L}_i(\theta) = \log P(y_i \mid x_i; \theta) = y_i \log \sigma(\theta^T x_i) + (1-y_i) \log \left(1 - \sigma(\theta^T x_i)\right) \]

We want to maximize log-likelihood. Equivalently (since optimization code typically minimizes), we define the logistic loss / Binary Cross-Entropy Loss as negative log-likelihood:

\[ J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \left[\ y_i \log h_\theta(x_i) + (1-y_i) \log \left(1 - h_\theta(x_i)\right)\ \right] \]

where \( h_\theta(x_i) = \sigma(\theta^T x_i) \).

Why this loss is well-behaved

2.5 Deriving the Gradient for Logistic Regression

The derivation is worth following once. Despite the sigmoid and the logarithms in the loss, the algebra simplifies to a gradient with exactly the same form as the one we obtained for linear regression. Only the hypothesis \( h \) differs.

Step 1: Derivative of one example's loss w.r.t. parameter \( \theta_j \):

\[ \frac{\partial J_i}{\partial \theta_j} = -\left( y_i \frac{1}{h} \frac{\partial h}{\partial \theta_j} - (1-y_i) \frac{1}{1-h} \frac{\partial h}{\partial \theta_j} \right) \]

Step 2: Combine fractions, cancel the \( h(1-h) \) denominator:

\[ \frac{\partial J_i}{\partial \theta_j} = -\frac{y_i(1-h) - (1-y_i)h}{h(1-h)} \cdot \frac{\partial h}{\partial \theta_j} = \frac{h - y_i}{h(1-h)} \cdot \frac{\partial h}{\partial \theta_j} \]

Step 3: Use the sigmoid derivative identity:

\[ \frac{\partial h}{\partial \theta_j} = h_\theta(x_i) \cdot (1 - h_\theta(x_i)) \cdot x_{i,j} \]

Substitute in — the \( h(1-h) \) factors cancel perfectly:

\[ \frac{\partial J_i}{\partial \theta_j} = (h_\theta(x_i) - y_i) \cdot x_{i,j} \]

Average over all m examples to get the full batch gradient:

\[ \frac{\partial J}{\partial \theta_j} = \frac{1}{m} \sum_{i=1}^{m} (h_\theta(x_i) - y_i) \cdot x_{i,j} \]

Final GD Update Rules (Logistic Regression)

These look identical in form to Linear Regression — the only difference is the hypothesis \( h \):

\begin{align} \theta_0 &:= \theta_0 - \alpha \cdot \frac{1}{m} \sum_{i=1}^{m} (h_\theta(x_i) - y_i) \cdot 1 \\ \theta_j &:= \theta_j - \alpha \cdot \frac{1}{m} \sum_{i=1}^{m} (h_\theta(x_i) - y_i) \cdot x_{i,j} \quad (\text{for } j \ge 1) \end{align}

Or, compactly using the design matrix \( X \):

\[ \theta := \theta - \frac{\alpha}{m} X^T \left( \sigma(X\theta) - y \right) \]

Python Code Sketch

Logistic Regression Training Pipeline A clean process diagram showing the sigmoid function, cost computation, and gradient descent optimization loop. Logistic Regression Training Pipeline From probability estimation to iterative parameter optimization INPUTS X, y, θ training data and parameters ACTIVATION sigmoid(z) return 1 / (1 + np.exp(-z)) maps model scores to probabilities OBJECTIVE EVALUATION compute_cost(X, y, θ) m = len(y) h = sigmoid(X.dot(θ)) cost = (-1/m) * (y.T.dot(log(h)) + (1-y).T.dot(log(1-h))) return cost loss metric OPTIMIZATION LOOP gradient_descent(X, y, θ, α, iters) m = len(y) cost_history = np.zeros(iters) for i in range(iters): h = sigmoid(X.dot(θ)) grad = X.T.dot(h - y) / m θ -= α * grad cost_history[i] = compute_cost(...) return θ, cost_history repeat for iters θ, cost_history trained parameters and loss trace data h prediction cost update loop s*

3. Interactive Examples

Example 1: Sigmoid Endpoints

Without a calculator, mentally approximate these sigmoid values:

A. \( \sigma(z) \) as \( z \to +\infty \) ?

Approaches \( 1 \). For large positive z, \( e^{-z} \to 0 \), so denominator → 1+0 = 1. In classification terms: very confident "positive class" prediction.

B. \( \sigma(z) \) as \( z \to -\infty \) ?

Approaches \( 0 \). For large negative z, \( e^{-z} = e^{|z|} \to \infty \), so denominator → ∞. In classification terms: very confident "negative class" prediction.

C. \( \sigma(0) \) ? What does this mean for a decision threshold at 0.5?

\( \sigma(0) = 1 / (1 + e^{0}) = 1/2 = 0.5 \). This tells us that the natural classification boundary (p = 0.5) corresponds to the linear predictor \( z = \theta^T x = 0 \). In other words, the decision boundary of logistic regression is the hyperplane \( \theta^T x = 0 \).

Example 2: Interpret the Coefficient Signs

You are building a churn model for a telecom (Churn = Yes/No). The learned coefficients are:

Question: For each feature, does increasing it make churn MORE likely or LESS likely? (Intuition only — no exact numbers needed.)

  • Monthly Charge (+): Higher monthly bills → higher log-odds of churn → MORE likely to churn.
  • Tenure (−): Longer-tenured customers → lower log-odds → LESS likely to churn (brand loyalty effect).
  • Tech Support (−): Customers who have tech support → significantly lower log-odds → MUCH less likely to churn (biggest magnitude of the three).

Business takeaway: Offering free/cheap tech support to at-risk high-charge new customers could reduce churn!

Example 3: Cross-Entropy Sanity Check

True label y = 1. Model A predicts p = 0.98. Model B predicts p = 0.45. Which has lower cross-entropy loss for this example, and by how much (roughly)?

Loss for y=1 is just \( -\log(p) \).
  • Model A: −log(0.98) ≈ 0.02 (low loss — correct, confident prediction).
  • Model B: −log(0.45) ≈ 0.80 (18× higher loss — uncertain on the correct class).

Model A wins by a mile. Cross-entropy heavily penalizes confident-but-wrong predictions and also uncertain predictions on clear examples.

4. Numerical Solutions

Problem 1: Compute Sigmoid + Class Prediction

Logistic model: \( z = -3 + 1.5 x_1 - 0.2 x_2 \). Customer features: \( x_1 = 4 \), \( x_2 = 5 \).

📘 Step-by-Step

Step 1: Compute linear predictor z:

\[ z = -3 + 1.5(4) - 0.2(5) = -3 + 6 - 1 = \mathbf{2} \]

Step 2: Apply sigmoid:

\[ p = \sigma(2) = \frac{1}{1 + e^{-2}} \approx \frac{1}{1 + 0.1353} \approx \mathbf{0.881} \]

Step 3: Classify with threshold 0.5:

0.881 > 0.5 → Predicted class = 1 (positive).

Problem 2: Binary Cross-Entropy for a Mini-Batch

Batch of 3 examples. Compute the total average loss \( J(\theta) \).

i\( y_i \)\( p_i = \sigma(z_i) \)\( y_i \log p_i \)\( (1-y_i)\log(1-p_i) \)Sum = loss contribution
110.90log 0.9 ≈ −0.1050−0.105
200.600log 0.4 ≈ −0.916−0.916
310.30log 0.3 ≈ −1.2040−1.204
📘 Compute J(θ)

Step 1: Sum the last column: −0.105 − 0.916 − 1.204 = −2.225.

Step 2: Apply \( J = -\frac{1}{m} \sum \):

\[ J(\theta) = -\frac{1}{3} \times (-2.225) = \frac{2.225}{3} \approx \mathbf{0.742} \]

Interpretation: Example 3 (y = 1 but p only 0.30) contributes the most loss — the model confidently got it wrong! That's the point of cross-entropy: it punishes bad calls.

Problem 3: Single-Step Gradient Descent Update

One example (m = 1): \( x_0 = 1,\ x_1 = 2 \). True \( y = 1 \). Current \( \theta_0 = 0 \), \( \theta_1 = 0 \), α = 0.5.

📘 Full Step-by-Step GD Update

Step 1: Compute z and \( h = \sigma(z) \):

\[ z = 0 \cdot 1 + 0 \cdot 2 = 0 \implies h = \sigma(0) = 0.5 \]

Step 2: Error \( h - y = 0.5 - 1 = -0.5 \).

Step 3: Gradients (divide by m = 1):

\begin{align} \frac{\partial J}{\partial \theta_0} &= (-0.5) \cdot x_0 = -0.5 \cdot 1 = -0.5 \\ \frac{\partial J}{\partial \theta_1} &= (-0.5) \cdot x_1 = -0.5 \cdot 2 = -1.0 \end{align}

Step 4: Apply the updates (θ = θ − α · gradient):

\begin{align} \theta_0 &:= 0 - 0.5(-0.5) = \mathbf{+0.25} \\ \theta_1 &:= 0 - 0.5(-1.0) = \mathbf{+0.5} \end{align}

Sanity check: both parameters moved in the positive direction, which increases \( z = \theta^T x \), which increases \( \sigma(z) \), which moves p upward toward the true label y = 1. ✓ (The model was under-confident in the positive class; it corrected itself.)

5. Try It Yourself

Problem 1 — Convert Log-Odds back to Probability

A logistic regression predicts log-odds \( \ln(p/(1-p)) = -1 \) for a given customer. What is the predicted probability p? What class is predicted at the 0.5 threshold?

Exponentiate: \( p/(1-p) = e^{-1} \approx 0.368 \).

Solve: \( p = 0.368 (1-p) \implies p + 0.368p = 0.368 \implies p = 0.368/1.368 \approx \mathbf{0.269} \).

p ≈ 26.9% < 50% → Predicted class = 0 (negative class).

Problem 2 — Why Not Squared Error?

A student suggests: "Why not just use MSE on top of the sigmoid? That way we don't have to derive all this log stuff." Give two reasons (statistical or optimization-based) why binary cross-entropy is the better choice for logistic regression.

  1. Statistical: Cross-entropy = negative log-likelihood of the Bernoulli model. Maximizing likelihood gives consistent, efficient estimators (the classic "maximum likelihood" theory guarantees good asymptotic properties). MSE has no such justification for Bernoulli data.
  2. Optimization: Cross-entropy loss is convex in θ and yields clean, simple gradients (the h-y form you derived). MSE on top of sigmoid produces a non-convex loss with potential local minima and more complicated gradients that suffer from vanishing gradients more severely.
  3. (Bonus) Practical: Cross-entropy heavily penalizes confident wrong predictions, which matches our intuition for good classification. MSE under-penalizes those extreme errors.
Problem 3 — One More GD Step

After the update in Problem 3 (Numerical Solutions), you have new parameters \( \theta_0 = 0.25 \), \( \theta_1 = 0.5 \). The same example (\( x_0=1, x_1=2, y=1 \)) is processed again in the next iteration. Compute the NEW gradients \( \partial J/\partial \theta_0 \) and \( \partial J/\partial \theta_1 \).

z = 0.25 + 0.5(2) = 1.25.

h = σ(1.25) = 1 / (1 + e−1.25) ≈ 1 / (1 + 0.287) ≈ 0.777.

Error = h − y = 0.777 − 1.000 = −0.223 (notice it's smaller than the first iteration's −0.5! The model is learning.)

\begin{align} \frac{\partial J}{\partial \theta_0} &= \frac{1}{1}(-0.223) \cdot 1 = \mathbf{-0.223} \\ \frac{\partial J}{\partial \theta_1} &= \frac{1}{1}(-0.223) \cdot 2 = \mathbf{-0.446} \end{align}

Gradients got smaller — the model is converging toward the correct answer. This is what healthy GD looks like.

6. Interactive Quiz

Answer all 5 questions. Click an option for instant feedback.

Your score: 0 / 5

7. Key Takeaways

  1. Linear regression is for regression only. Classification needs outputs ∈ [0, 1] and a proper likelihood — enter logistic regression.
  2. Logistic = linear regression on the log-odds. log(p/(1−p)) = θᵀx. Invert with sigmoid: p = σ(θᵀx) = 1 / (1 + e−θᵀˣ).
  3. Sigmoid shape: σ(−∞)=0, σ(0)=0.5, σ(+∞)=1. Decision boundary is θᵀx = 0 (linear — LR is a linear classifier!).
  4. Binary Cross-Entropy loss = −log-likelihood of a Bernoulli: J = −(1/m)Σ [y log p + (1−y) log(1−p)]. Convex, statistically justified, nice gradients.
  5. Gradient update is elegant: ∂J/∂θⱼ = (1/m)Σ (h − y)·xⱼ. Same form as linear regression! Only the hypothesis h changes (sigmoid vs. linear).
  6. Interpret coefficients carefully: sign(θⱼ) tells you direction (+ raises, − lowers P(Y=1)); magnitude times log(e) = multiplicative change in odds ratio per unit increase in xⱼ.

8. Common Pitfalls

  1. Interpreting logistic coefficients as "change in probability per x." Coefficients are linear in log-odds, not linear in probability! The effect on probability is non-linear — it depends on where you are on the sigmoid (bigger around 0.5, smaller near the extremes).
  2. Using MSE loss with sigmoid outputs. MSE on top of sigmoid is non-convex, gradient-unfriendly, and statistically unjustified. Binary cross-entropy is the correct, industry-standard loss.
  3. Reporting σ(z) as a "probability" without calibration checks. LR probabilities are usually well-calibrated (it's a likelihood model), but if you've added heavy regularization or used weird thresholds, always check a reliability diagram / calibration curve before trusting them for decision making.
  4. Calling logistic regression a "regression" algorithm and using it for numerical prediction. Despite the name, it's a CLASSIFIER. The output is a class probability, not an unbounded numerical forecast. Use linear regression for real-valued targets.
  5. Hard-thresholding at 0.5 for imbalanced tasks. At 9.6% loan acceptance, the default 0.5 threshold predicts "Not Accept" for almost everyone. Tune the threshold (e.g., on precision-recall curve) to trade precision for recall based on business costs.
  6. Forgetting feature scaling before regularized logistic regression. L2 penalty in sklearn's LogisticRegression is on by default (C inverse of λ). Without standardization, features with large units get penalized way more heavily.